Noise Mitigation for a Voice Interface Device

ABSTRACT

A method at an electronic device with one or more microphones and a speaker, the electronic device configured to be responsive to any of a plurality of affordances including a voice-based affordance, includes determining background noise of an environment associated with the electronic device, and before detecting the voice-based affordance: determining whether the background noise would interfere with recognition of the hotword in voice inputs detected by the electronic device, and if so, indicating to a user to use an affordance other than the voice-based affordance.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/367,174, filed Mar. 27, 2019, which is a continuation of U.S.application Ser. No. 15/284,493, filed Oct. 3, 2016 and issued as U.S.Pat. No. 10,283,138 on May 7, 209, each of which is hereby incorporatedby reference in its entirety.

This application is related to U.S. Provisional Patent Application No.62/334,434, entitled “Implementations for Voice Assistant on Devices,”filed May 10, 2016; and U.S. Provisional Patent Application No.62/336,566, entitled “LED Design Language for Visual Affordance of VoiceUser Interfaces,” filed May 13, 2016, each of which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

The disclosed implementations relates generally to voice interfaces andrelated devices, including but not limited to methods and systems fornoise mitigation for a voice interface device.

BACKGROUND

Electronic devices integrated with microphones have been widely used tocollect voice inputs from users and implement voice-activated functionsaccording to the voice inputs. For example, many state-of-the-art mobiledevices include a voice assistant feature (e.g., Siri, Google Assistant)that is configured to use voice inputs to initiate a phone call, conducta restaurant search, start routing on a map, create calendar events, adda post to a social network, recognize a song, and complete many othertasks.

A location (e.g., a room or space within a home) may include multipledevices that include voice assistant systems and one or more users ofsuch devices. As the devices are more than ever configured to be capableof accessing and presenting user-specific information, it is desirablefor a voice assistant device to be able to handle multiple usersgracefully and be able to provide user-specific information to theappropriate user. It is also desirable for there to be a leader amongstthe voice assistant devices that is responsible for responding to theuser's voice inputs, in order to reduce user confusion. Further, it isdesirable for a voice assistant device to include noise mitigationmeasures in case the voice assistant devices are located at a noisylocation.

SUMMARY

Accordingly, there is a need for an electronic device with a voiceassistant system and/or a voice assistant server system thatincorporates methods and systems for handling multiple users, formulti-device coordination to determine a leader, and for noisemitigation. The device and/or the server system may be configured toidentify a speaker of a voice input based on comparisons to voicemodels, and to personalize responses to the identified speaker. Thedevice and/or the server system may also be configured to negotiateleadership amongst multiple devices for response to a speaker. Thedevice and/or the server system may further be configured to detect thatan environment around the device is sufficiently noisy where analternative method, to a hotword voice input, of waking up the device isadvisable.

In accordance with some implementations, a method includes, at anelectronic device with one or more microphones, a speaker, one or moreprocessors, and memory storing one or more programs for execution by theone or more processors: receiving a first freeform voice input,comparing the first voice input to a set of voice models associated witha plurality of users associated with the electronic device, where thevoice models are trained to the electronic device, based on thecomparing, determining that the first voice input corresponds to a firstuser of the plurality of users, and presenting a response in accordancewith the determination.

In some implementations, an electronic device includes one or moremicrophones, a speaker, one or more processors, and memory storing oneor more programs for execution by the one or more processors. The one ormore programs include instructions for: receiving a first freeform voiceinput, comparing the first voice input to a set of voice modelsassociated with a plurality of users associated with the electronicdevice, where the voice models are trained to the electronic device,based on the comparing, determining that the first voice inputcorresponds to a first user of the plurality of users, and presenting aresponse in accordance with the determination.

In some implementations, a non-transitory computer readable storagemedium stores one or more programs. The one or more programs includeinstructions, which, when executed by an electronic device with one ormore microphones, a speaker, and one or more processors, cause theelectronic device to perform operations including: receiving a firstfreeform voice input, comparing the first voice input to a set of voicemodels associated with a plurality of users associated with theelectronic device, where the voice models are trained to the electronicdevice, based on the comparing, determining that the first voice inputcorresponds to a first user of the plurality of users, and presenting aresponse in accordance with the determination.

In accordance with some implementations, a method includes, at a firstelectronic device of a plurality of electronic devices, each electronicdevice of the plurality of electronic devices comprising one or moremicrophones, a speaker, one or more processors, and memory storing oneor more programs for execution by the one or more processors: detectinga voice input, determining a quality score for the detected voice input,communicating the quality score to the other devices of the plurality ofelectronic devices, receiving quality scores generated by the otherdevices of the plurality of electronic devices for detection of thevoice input by the other devices, in accordance with a determinationthat the quality score generated by the first electronic device is thehighest amongst the generated quality score and received quality scoresfor the voice input, outputting an audible and/or a visual response tothe detected voice input, where the other devices of the plurality ofelectronic devices forgo outputting an audible response to the detectedvoice input, and in accordance with a determination that the qualityscore generated by the first electronic device is not the highestamongst the quality scores for the voice input generated by theplurality of electronic devices, forgoing outputting a response to thedetected voice input.

In accordance with some implementations, a first electronic device of aplurality of electronic devices includes, and each of the plurality ofelectronic devices includes, one or more microphones, a speaker, one ormore processors, and memory storing one or more programs to be executedby the one or more processors. The one or more programs includeinstructions for: detecting a voice input; determining a quality scorefor the detected voice input; communicating the quality score to theother devices of the plurality of electronic devices; receiving qualityscores generated by the other devices of the plurality of electronicdevices for detection of the voice input by the other devices; inaccordance with a determination that the quality score generated by thefirst electronic device is the highest amongst the generated qualityscore and received quality scores for the voice input, outputting anaudible and/or a visual response to the detected voice input, where theother devices of the plurality of electronic devices forgo outputting anaudible response to the detected voice input; and in accordance with adetermination that the quality score generated by the first electronicdevice is not the highest amongst the quality scores for the voice inputgenerated by the plurality of electronic devices, forgoing outputting aresponse to the detected voice input.

In accordance with some implementations, a non-transitory computerreadable storage medium stores one or more programs. The one or moreprograms include instructions, which, when executed by a firstelectronic device of a plurality of electronic devices, each of theplurality of electronic device comprising one or more microphones, aspeaker, and one or more processors, cause the electronic device toperform operations including: detecting a voice input; determining aquality score for the detected voice input; communicating the qualityscore to the other devices of the plurality of electronic devices;receiving quality scores generated by the other devices of the pluralityof electronic devices for detection of the voice input by the otherdevices; in accordance with a determination that the quality scoregenerated by the first electronic device is the highest amongst thegenerated quality score and received quality scores for the voice input,outputting an audible and/or a visual response to the detected voiceinput, where the other devices of the plurality of electronic devicesforgo outputting an audible response to the detected voice input; and inaccordance with a determination that the quality score generated by thefirst electronic device is not the highest amongst the quality scoresfor the voice input generated by the plurality of electronic devices,forgoing outputting a response to the detected voice input.

In some implementations, a method includes, at an electronic device withone or more microphones, a speaker, one or more processors, and memorystoring one or more programs for execution by the one or moreprocessors, the electronic device configured to be awakened by any of aplurality of affordances including a voice-based affordance: determininga noise profile of an environment around the electronic device;determining whether the noise profile interferes with the voice-basedaffordance; and in accordance with a determination that the noiseprofile interferes with the voice-based affordance, presenting a hint toa user to use an affordance of the plurality of affordances other thanthe voice-based affordance to awaken the electronic device.

In some implementations, an electronic device includes one or moremicrophones, a speaker, one or more processors, and memory storing oneor more programs to be executed by the one or more processors. Theelectronic device is configured to be awakened by any of a plurality ofaffordances including a voice-based affordance. The one or more programsinclude instructions for: determining a noise profile of an environmentaround the electronic device, determining whether the noise profileinterferes with the voice-based affordance, and in accordance with adetermination that the noise profile interferes with the voice-basedaffordance, presenting a hint to a user to use an affordance of theplurality of affordances other than the voice-based affordance to awakenthe electronic device.

A non-transitory computer readable storage medium stores one or moreprograms. The one or more programs include instructions, which, whenexecuted by an electronic device with one or more microphones, aspeaker, and one or more processors, the electronic device configured tobe awakened by any of a plurality of affordances including a voice-basedaffordance, cause the electronic device to perform operations including:determining a noise profile of an environment around the electronicdevice, determining whether the noise profile interferes with thevoice-based affordance, and in accordance with a determination that thenoise profile interferes with the voice-based affordance, presenting ahint to a user to use an affordance of the plurality of affordancesother than the voice-based affordance to awaken the electronic device.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described implementations,reference should be made to the Description of Implementations below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is an example operating environment in accordance with someimplementations.

FIGS. 2A-2B are block diagrams illustrating an example electronic devicethat is applied as a voice interface to collect user voice commands inan operating environment in accordance with some implementations.

FIG. 3 is a block diagram illustrating an example server in the serversystem of an operating environment in accordance with someimplementations. An example server is one of a voice assistance server.

FIGS. 4A-4B are diagrams illustrating an example device leadershipnegotiation, in accordance with some implementations.

FIG. 5 is a flow diagram illustrating a method of responding to a voiceinput of a user in accordance with some implementations.

FIG. 6 is a flow diagram illustrating a method of negotiating deviceleadership among multiple voice interface devices in accordance withsome implementations.

FIG. 7 is a flow diagram illustrating a method of mitigating noisearound a voice interface device in accordance with some implementations.

FIGS. 8A and 8B are a front view and a rear view of a voice-activatedelectronic device in accordance with some implementations.

FIG. 8C is a top view of a voice-activated electronic device inaccordance with some implementations, and FIG. 8D shows six visualpatterns displayed by an array of full color LEDs for indicating voiceprocessing states in accordance with some implementations.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DESCRIPTION OF IMPLEMENTATIONS

While the digital revolution has provided many benefits ranging fromopenly sharing information to a sense of global community, emerging newtechnology often induces confusion, skepticism and fear among consumers,preventing consumers from benefitting from the technology. Electronicdevices are conveniently used as voice interfaces to receive voiceinputs from users and initiate voice-activated functions, and therebyoffer eyes-free and hands-free solutions to approach both existing andemerging technology. Specifically, the voice inputs received at theelectronic device can carry instructions and information even if auser's line of sight is obscured and his hands are full. To enable ahands-free and eyes-free experience, the voice-activated electronicdevice listens to the ambient (i.e., constantly processes audio signalscollected from the ambient) constantly or only when triggered. On theother hand, user identities are linked with a user's voice and alanguage used by the user. To protect the user identities, thesevoice-activated electronic devices are normally used in non-publicplaces that are protected, controlled and intimate spaces (e.g., homeand car).

In accordance with some implementations, a voice-activated electronicdevice identifies a speaker of a voice input as one of one or moreassociated or registered users. Responses to the speaker output by theelectronic device are personalized to the identified speaker. Thespeaker is identified by comparing the speaker's voice input to a set oftrained voice models or voice fingerprints. In some implementations,words and phrases used by a speaker to train the voice models are notnecessarily predefined and do not need to match words later spoken bythe speaker in order for the electronic device to identify the speaker.

Further, in accordance with some implementations, multiplevoice-activated electronic devices negotiate a leader amongst themselvesto respond to voice inputs from a user. The negotiation may be based onwhich device detected the voice input the best, or which device is theclosest to the user. Additionally, if the voice input is particularlyrelevant to one of the devices, that device may be selected to respondeven if it would otherwise not be selected for response; the relevanceof the command included in the voice input to a particular device (e.g.,“stop music” should refer to a device playing music) and the state ofthe device receiving the voice input (e.g., “screen on” vs. “screenoff”) are considered in determining which device will be the leader.

Further, in accordance with some implementations, a voice-activateddevice determines whether the surrounding noise is sufficientlyinterfering with detection of spoken hotwords for waking up the device(in some implementations, a “hotword” is a user defined or predefinedword or phrase used to “wake-up” or trigger a voice-activated device toattend/respond to a spoken command that is issued subsequent to thehotword) or other spoken voice inputs. If the noise is sufficientlyinterfering, the device indicates such and gives the user a hint to usean alternative way of waking up the device or other noise mitigationmeasures.

Voice Assistant Operating Environment

FIG. 1 is an example operating environment in accordance with someimplementations. Operating environment 100 includes one or morevoice-activated electronic devices 190 (e.g., electronic devices 190-1thru 190-N). The one or more voice-activated electronic devices 190 maybe located in one or more locations (e.g., all in a room or space of astructure, spread out throughout multiple spaces within a structure orthroughout multiple structures (e.g., one in a house and one in theuser's car)). The environment 100 optionally includes one or moredevices (e.g., media devices, smart home devices) (not shown) with whichthe voice-activated electronic devices 190 can communicate.

The electronic devices 190 are communicatively coupled, throughcommunication networks 110, to a voice assistance server 112 of a voiceassistant service. One or more of the electronic devices 190 arecommunicatively coupled to a local network 104, which is communicativelycoupled to the communication networks 110 (e.g., electronic devices190-1 and 190-2). In some implementations, the local network 104 is alocal area network implemented at a network interface (e.g., a router).The electronic devices 190 that are communicatively coupled to the localnetwork 104 may also communicate with each other through the localnetwork 104.

Optionally, one or more of the electronic devices 190 arecommunicatively coupled to the communication networks 110 and are not onthe local network 104 (e.g., electronic device 190-N). For example,these electronic devices 190 are not on the Wi-Fi network correspondingto the local network 104 but are connected to the communication networks110 through a cellular connection. In some implementations,communication between electronic devices 190 that are on the localnetwork 104 and electronic devices 190 that are not on the local network104 are done through the voice assistance server 112. The electronicdevices 190 are registered in a device registry 118 of the voiceassistant service and thus known to the voice assistance server 112.

The environment 100 also includes one or more content hosts 114. Acontent host 114 may be a remote content source from which content isstreamed or otherwise obtained in accordance with a user voice request.A content host 114 may be an information source from which the voiceassistance server 112 retrieves information in accordance with a uservoice request.

In some implementations, an electronic device 190 may be associated withmultiple users having respective user accounts in the user domain. Anyof these users, as well as users not associated with the device, maymake voice inputs to the electronic device 190. The electronic device190 receives these voice inputs from these users 102-1 thru 102-M(including associated and non-associated users), and the electronicdevice 190 and/or the voice assistance server 112 proceeds to identify,for a voice input, the user making the voice input. With the useridentification, a response to that voice input may be personalized tothe identified user.

In some implementations, the environment 100 includes multipleelectronic devices 190 (e.g., devices 190-1 thru 190-N). The devices 190are located throughout the environment 100 (e.g., all within a room orspace in a structure, spread throughout the structure, some within thestructure and some without). When a user 102 makes a voice input, eachof the devices 190 either receives the input or does not receive theinput (e.g., if the device was too far away from the user). The devicesthat receive the input receive the input at varying degrees of quality;the quality of the sample of the voice input at a device is based onmultiple factors, including but not limited to distance of the user fromthe device and the noise around the device. The multiple devices 190negotiate a leader amongst themselves to respond to the user and toreceive further voice input from the user 102 based on the quality ofthe samples of the voice inputs.

In some implementations, an electronic device 190 determines a level ofnoise around the device and determines whether the determined noiselevel is sufficiently high to interfere with recognition of the hotwordin voice inputs, and thus interfere with awakening of the device byvoice, or with recognition of voice requests. If the noise level isdetermined to be sufficiently high to be interfering, the electronicdevice 190 indicates to the user that the noise level is interfering andgives the user a hint that the user should use another way to wake upthe electronic device 190 (e.g., activate a button). The indication ofthe interfering noise level and the hint to use another way to wake upthe device may be done in the same presentation (e.g., illuminating thewake-up button).

In some implementations, one or more media devices are disposed in theoperating environment 100 to provide to one or more occupants mediacontent, news and/or other information. In some implementations, thecontent provided by the media devices is stored at a local contentsource, streamed from a remote content source (e.g., content host(s)114), or generated locally (e.g., through a local text to voiceprocessor that reads a customized news briefing, emails, texts, a localweather report, etc. to one or more occupants of the operatingenvironment 100). In some implementations, the media devices includemedia output devices that directly output the media content to anaudience (e.g., one or more users), and cast devices that are networkedto stream media content to the media output devices. Examples of themedia output devices include, but are not limited to television (TV)display devices and music players. Examples of the cast devices include,but are not limited to, set-top boxes (STBs), DVD players, TV boxes, andmedia streaming devices, such as Google's Chromecast™ media streamingdevice.

In some implementations, the media devices include one or morevoice-activated electronic devices 190 that receive, process and respondto voice commands of occupants. In some implementations, thevoice-activated electronic devices 190 respond to voice commands by:generating and providing a spoken response to a voice command (e.g.,speaking the current time in response to the question, “what time isit?”); streaming media content requested by a user (e.g., “play a BeachBoys song”); reading a news story or a daily news briefing prepared forthe user; playing a media item stored on the personal assistant deviceor on the local network; changing a state or operating one or more otherconnected devices within the operating environment 100 (e.g., turninglights, appliances or media devices on/off, locking/unlocking a lock,opening windows, etc.); or issuing a corresponding request to a servervia a network 110.

In some implementations, the one or more electronic devices 190 aredisposed in the operating environment 100 to collect audio inputs forinitiating various functions, including media play functions of themedia devices. In some implementations, these voice-activated electronicdevices 190 (e.g., devices 190-1 thru 190-N) are disposed in proximityto a media device, for example, in the same room with the cast devicesand the media output devices. Alternatively, in some implementations, avoice-activated electronic device 190 is disposed in a room having oneor more smart home devices but not any media device. Alternatively, insome implementations, a voice-activated electronic device 190 isdisposed in a location having no networked electronic device. Further,in some implementations, a room or space in the structure may havemultiple electronic devices 190.

In some implementations, the electronic device 190 includes at least oneor more microphones, a speaker, a processor and memory storing at leastone program for execution by the processor. The speaker is configured toallow the electronic device 190 to deliver voice messages and otheraudio (e.g., audible tones) to a location where the electronic device190 is located in the operating environment 100, thereby broadcastingmusic, reporting a state of audio input processing, having aconversation with or giving instructions to a user of the electronicdevice 190. As an alternative to the voice messages, visual signalscould also be used to provide feedback to the user of the electronicdevice 190 concerning the state of audio input processing. When theelectronic device 190 is a mobile device (e.g., a mobile phone or atablet computer), its display screen is configured to display anotification concerning the state of audio input processing.

In some implementations, the electronic device 190 is a voice interfacedevice that is network-connected to provide voice recognition functionswith the aid of a voice assistance server 112 and/or optionally a cloudcast service server (not shown). For example, the electronic device 190includes a smart speaker that provides music to a user and allowseyes-free and hands-free access to a voice assistant service (e.g.,Google Assistant). Optionally, the electronic device 190 is one of adesktop or laptop computer, a tablet, a mobile phone that includes amicrophone, a cast device that includes a microphone and optionally aspeaker, an audio system (e.g., a stereo system, a speaker system, aportable speaker) that includes a microphone and a speaker, a televisionthat includes a microphone and a speaker, and a user interface system inan automobile that includes a microphone and a speaker and optionally adisplay. Optionally, the electronic device 190 is a simple and low costvoice interface device. Generally, the electronic device 190 may be anydevice that is capable of network connection and that includes amicrophone, a speaker, and programs, modules, and data for interactingwith voice assistant service. Given simplicity and low cost of theelectronic device 190, the electronic device 190 includes an array oflight emitting diodes (LEDs) rather than a full display screen, anddisplays a visual pattern on the LEDs to indicate the state of audioinput processing. In some implementations, the LEDs are full color LEDs,and the colors of the LEDs may be employed as a part of the visualpattern to be displayed on the LEDs. Multiple examples of using LEDs todisplay visual patterns in order to convey information or device statusare described in U.S. Provisional Patent Application No. 62/336,566,entitled “LED Design Language for Visual Affordance of Voice UserInterfaces,” filed May 13, 2016, which is incorporated by reference inits entirety. In some implementations, visual patterns indicating thestate of voice processing operations are displayed using characteristicimages shown on conventional displays associated with electronic devicesthat are performing the voice processing operations.

In some implementations, LEDs or other visual displays are used toconvey a collective voice processing state of multiple participatingelectronic devices. For example, in an operating environment where thereare multiple voice processing or voice interface devices (e.g., multipleelectronic devices 400 as shown in FIG. 4A of the '566 application,duplicated as FIG. 8A below; multiple electronic devices 190), groups ofcolor LEDs (e.g., LEDs 404 as shown in FIG. 4A of the '566 application)associated with respective electronic devices can be used to conveywhich of the electronic devices is listening to a user, and which of thelistening devices is the leader (where the “leader” device generallytakes the lead in responding to a spoken request issued by the user).

More generally, the '566 application describes (e.g., see paras.[0087]-[0100]) a “LED Design Language” for indicating visually using acollection of LEDs a variety of voice processing states of an electronicdevice, such as a “Hot word detection state and listening state,” a“Thinking mode or working mode,” and a “Responding mode or speakingmode.” In some implementations, unique states of voice processingoperations described herein are represented using a group of LEDs inaccordance with one or more aspects of the “LED Design Language” of the'566 application. These visual indicators can also be combined with oneor more audible indicators generated by electronic devices that areperforming voice processing operations. The resulting audio and/orvisual indicators will enable users in a voice-interactive environmentto understand the state of various voice processing electronic devicesin the environment and to effectively interact with those devices in anatural, intuitive manner.

When voice inputs from the electronic device 190 are used to control themedia output devices via the cast devices, the electronic device 190effectively enables a new level of control of cast-enabled mediadevices. In a specific example, the electronic device 190 includes acasual enjoyment speaker with far-field voice access and functions as avoice interface device for the voice assistant service. The electronicdevice 190 could be disposed in any area in the operating environment100. When multiple electronic devices 190 are distributed in multiplerooms, they become cast audio receivers that are synchronized to providevoice inputs from all these rooms.

Specifically, in some implementations, the electronic device 190includes a Wi-Fi speaker with a microphone that is connected to avoice-activated voice assistant service (e.g., Google Assistant). A usercan issue a media play request via the microphone of electronic device190, and ask the voice assistant service to play media content on theelectronic device 190 itself or on another connected media outputdevice. For example, the user can issue a media play request by sayingto the Wi-Fi speaker “OK Google, play cat videos on my Living room TV.”The voice assistant service then fulfils the media play request byplaying the requested media content on the requested device using adefault or designated media application.

In some implementations, a user can issue a voice request, via themicrophone of the electronic device 190, concerning media content thathas already been played or is being played on a display device (e.g.,the user can ask for information about the media content, buy the mediacontent through an online store, or compose and issue a social postabout the media content).

In some implementations, a user may want to take a current media sessionwith them as they move through the house and can request such a servicefrom one or more of the electronic devices 190. This requires the voiceassistant service to transfer the current media session from a firstcast device to a second cast device that is not directly connected tothe first cast device or has no knowledge of the existence of the firstcast device. Subsequent to the media content transfer, a second outputdevice coupled to the second cast device continues to play the mediacontent previously a first output device coupled to the first castdevice from the exact point within a music track or a video clip whereplay of the media content was forgone on the first output device. Insome implementations, the electronic device 190 that receives therequest to transfer the media session can satisfy the request. In someimplementations, the electronic device 190 that receives the request totransfer the media session relays the request to another device orsystem (e.g., a hub device, voice assistance server 112) for handling.

Further, in some implementations, a user may issue, via the microphoneof electronic device 190, a request for information or for performanceof an action or operation. The information requested may be personal(e.g., the user's emails, the user's calendar events, the user's flightinformation, etc.), non-personal (e.g., sports scores, news stories,etc.) or somewhere in between (e.g., scores for teams or sportspreferred by the user, news stories from the user's preferred sources,etc.). The requested information or action/operation may involve accessto personal information (e.g., purchasing a digital media item withpayment information provided by the user, purchasing a physical good).The electronic device 190 responds to the request with voice messageresponses to the user, where the response may include, for example,requests for additional information to fulfill the request, confirmationthat the request has been fulfilled, notice that the request cannot befulfilled, and so forth.

In some implementations, in addition to the voice-activated electronicdevices 190 and the media devices (e.g., the output devices and the castdevices), the operating environment 100 may also include one or moresmart home devices (not shown). The integrated smart home devicesinclude intelligent, multi-sensing, network-connected devices thatintegrate seamlessly with each other in a smart home network and/or witha central server or a cloud-computing system to provide a variety ofuseful smart home functions. In some implementations, a smart homedevice is disposed at the same location of the operating environment 100as a cast device and/or an output device, and therefore, is located inproximity to or with a known distance with respect to the cast deviceand the output device.

The smart home devices in the operating environment 100 may include, butare not limited to, one or more intelligent, multi-sensing,network-connected thermostats, one or more intelligent,network-connected, multi-sensing hazard detectors, one or moreintelligent, multi-sensing, network-connected entryway interface devicesand (hereinafter referred to as “smart doorbells” and “smart doorlocks”), one or more intelligent, multi-sensing, network-connected alarmsystems, one or more intelligent, multi-sensing, network-connectedcamera systems, and one or more intelligent, multi-sensing,network-connected wall switches, and one or more intelligent,multi-sensing, network-connected power sockets. In some implementations,the smart home devices in the operating environment 100 of FIG. 1includes a plurality of intelligent, multi-sensing, network-connectedappliances (hereinafter referred to as “smart appliances”), such asrefrigerators, stoves, ovens, televisions, washers, dryers, lights,stereos, intercom systems, garage-door openers, floor fans, ceilingfans, wall air conditioners, pool heaters, irrigation systems, securitysystems, space heaters, window AC units, motorized duct vents, and soforth. In some implementations, any one of these smart home device typescan be outfitted with microphones and one or more voice processingcapabilities as described herein so as to in whole or in part respond tovoice requests from an occupant or user.

In some implementations, each of the cast devices and thevoice-activated electronic devices 190 is capable of data communicationsand information sharing with other cast devices, voice-activatedelectronic devices 190, smart home devices, a central server orcloud-computing system, and/or other devices (e.g., the client device)that are network-connected. Data communications may be carried out usingany of a variety of custom or standard wireless protocols (e.g., IEEE802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart,ISA100.11a, WirelessHART, MiWi, etc.) and/or any of a variety of customor standard wired protocols (e.g., Ethernet, HomePlug, etc.), or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

Through the communication networks (e.g., the Internet) 110, the castdevices, the electronic devices 190 and the smart home devices maycommunicate with a server system (also called a central server systemand/or a cloud-computing system herein). Optionally, the server systemmay be associated with a manufacturer, support entity, or serviceprovider associated with the cast devices and the media contentdisplayed to the user. Accordingly, the server system includes the voiceassistance server 112 that processes audio inputs collected byvoice-activated electronic devices 190, one or more content hosts 114that provide the displayed media content, a cloud cast service servercreating a virtual user domain based on distributed device terminals,and the device registry 118 that keeps a record of the distributeddevice terminals in the virtual user environment. Examples of thedistributed device terminals include, but are not limited to the castdevices, the media output devices, the electronic devices 190 and thesmart home devices. In some implementations, these distributed deviceterminals are linked to a user account (e.g., a Google user account) inthe virtual user domain. It should be appreciated that processing ofaudio inputs collected by electronic devices 190 can be performedlocally at an electronic device 190, at a voice assistance server 112,at another smart home device (e.g., a hub device) or at some combinationof all or subset of the above.

It will be appreciated that in some implementations the electronicdevice(s) 190 also function in an environment without smart homedevices. For example, an electronic device 190 can, even in the absenceof smart home devices, respond to user requests for information orperformance of an action, and/or to initiate or control various mediaplay functions. An electronic device 190 can also function in a widerange of environments, including, without limitation, a vehicle, a ship,a business, or a manufacturing environment.

In some implementations, an electronic device 190 is “awakened” (e.g.,to activate an interface for the voice assistant service on theelectronic device 190, to put the electronic device 190 into a statewhere the electronic device 190 is ready to receive voice requests tothe voice assistant service) by a voice input that includes a hotword(also called a “wake word”). In some implementations, the electronicdevice 190 requires awakening if the electronic device 190 has been idlewith respect to receipt of voice inputs for at least a predefined amountof time (e.g., 5 minutes); the predefined amount of time corresponds toan amount of idle time allowed before a voice interface session orconversation times out. The hotword may be a word or phrase, and may bea predefined default and/or may be customized by a user (e.g., a usermay set a nickname for a particular electronic device 190 as thedevice's hotword). In some implementations, there may be multiplehotwords that can awaken an electronic device 190. A user may speak thehotword, wait for an acknowledgement response from the electronic device190 (e.g., the electronic device 190 outputs a greeting), and them makea first voice request. Alternatively, the user may combine the hotwordand the first voice request in one voice input (e.g., the voice inputincludes the hotword followed by the voice request).

In some implementations, the voice inputs spoken by the user to anelectronic device 190 may generally be freeform or natural languagespeech. That is, the voice input need not be strictly limited to apredefined set of words and phrases within a predefined syntax, withpossibly certain exceptions (e.g., user needs to speak the hotword firstto awaken the device).

In some implementations, an electronic device 190 includes one or moreadditional ways or affordances to wake up the device besides speakingthe hotword to the electronic device 190. The additional ways oraffordances may include, for example, activating a predefined button ortouch sensor (e.g., a touch sense array) on the electronic device 190.

In some implementations, a voice-activated electronic device 190interacts with a cast device, a client device or a server system of anoperating environment 100 in accordance with some implementations. Thevoice-activated electronic device 190 is configured to receive audioinputs from an environment in proximity to the voice-activatedelectronic device 190. Optionally, the electronic device 190 stores theaudio inputs and at least partially processes the audio inputs locally.Optionally, the electronic device 190 transmits the received audioinputs or the partially processed audio inputs to a voice assistanceserver 112 via the communication networks 110 for further processing.The cast device is configured to obtain media content or Internetcontent from one or more content hosts 114 for display on an outputdevice coupled to the cast device. In some implemenations, the castdevice and the voice-activated electronic device 190 are linked to eachother in a user domain, and more specifically, associated with eachother via a user account in the user domain. Information of the castdevice and information of the electronic device 190 are stored in thedevice registry 118 in association with the user account. In someimplementations, there is a device registry for cast devices and aregistry for voice-activated electronic devices 190. In someimplementations, a cloud cast service server manages the cast devicesregistry and the voice assistance server 112 manages the voice-activatedelectronic devices registry. The cast devices registry may referencedevices in the voice-activated electronic devices registry that areassociated in the user domain, and vice versa.

In some implementations, one or more of the electronic devices 190 (andone or more cast devices) are commissioned to the voice assistantservice via a client device (not shown). In some implementations, thevoice-activated electronic device 190 does not include any displayscreen, and relies on the client device to provide a user interfaceduring a commissioning process, and similarly for a cast device as well.Specifically, the client device is installed with an application thatenables a user interface to facilitate commissioning of a newvoice-activated electronic device 190 disposed in proximity to theclient device. A user may send a request on the user interface of theclient device to initiate a commissioning process for the new electronicdevice 190 that needs to be commissioned. After receiving thecommissioning request, the client device establishes a short rangecommunication link with the new electronic device 190 that needs to becommissioned. Optionally, the short range communication link isestablished based near field communication (NFC), Bluetooth, BluetoothLow Energy (BLE) and the like. The client device then conveys wirelessconfiguration data associated with a wireless local area network (WLAN)(e.g., local network 104) to the new or electronic device 190. Thewireless configuration data includes at least a WLAN security code(i.e., service set identifier (SSID) password), and optionally includesa SSID, an Internet protocol (IP) address, proxy configuration andgateway configuration. After receiving the wireless configuration datavia the short range communication link, the new electronic device 190decodes and recovers the wireless configuration data, and joins the WLANbased on the wireless configuration data.

In some implementations, additional user domain information is enteredon the user interface displayed on the client device, and used to linkthe new electronic device 190 to an account in a user domain.Optionally, the additional user domain information is conveyed to thenew electronic device 190 in conjunction with the wireless communicationdata via the short range communication link. Optionally, the additionaluser domain information is conveyed to the new electronic device 190 viathe WLAN after the new device has joined the WLAN.

Once the electronic device 190 has been commissioned into the userdomain, other devices and their associated activities may be controlledvia multiple control paths. In accordance with one control path, anapplication installed on the client device is used to control the otherdevice and its associated activities (e.g., media play activities).Alternatively, in accordance with another control path, the electronicdevice 190 is used to enable eyes-free and hands-free control of theother device and its associated activities.

Voice Activated Media Play on a Media Output Device

In some implementations, after the cast device and the voice-activatedelectronic device 190 are both commissioned and linked to a common userdomain, the voice-activated electronic device 190 can be used as a voiceuser interface to enable eyes-free and hands-free control of mediacontent streaming to the cast device involving no remote control, clientdevice or other second screen device. For example, the user may givevoice commands such as “Play Lady Gaga on Living Room speakers.” A LadyGaga music track or video clip is streamed to a cast device associatedwith the “Living Room speakers.” The client device is not involved, noris any cast device application or media play application loaded on theclient device.

In some implementations, a proxy service (e.g., voice assistant service,a cloud cast service) communicatively links the voice-activatedelectronic device 190 to the cast device and makes casting to the castdevice possible without involving any applications on the client device.Specifically, a voice message is captured and recorded by an electronicdevice 190, and the voice message is configured to request media play ona media output device. Optionally, the electronic device 190 partiallyprocesses the voice message locally. Optionally, the electronic device190 transmits the voice message or the partially processed voice messageto a voice assistance server 112 via the communication networks 110 forfurther processing. The voice assistance server 112 or a cloud castservice server determines that the voice message includes a first mediaplay request, and that the first media play request includes a uservoice command to play media content on a media output device and a uservoice designation of the media output device. The user voice commandfurther includes at least information of a first media play application(e.g., YouTube and Netflix) and the media content (e.g., Lady Gagamusic) that needs to be played.

In accordance with the voice designation of the media output device, thevoice assistance server 112 or the cloud cast service server identifiesin a device registry 118 a cast device associated in the user domainwith the electronic device 190 and coupled to the media output device.The cast device is configured to execute one or more media playapplications for controlling the media output device to play mediacontent received from one or more media content hosts 114. Then, thecloud cast service server sends to the cast device a second media playrequest including the information of the first media play applicationand the media content that needs to be played. Upon receiving theinformation sent by the cloud cast service server, the cast deviceexecutes the first media play application and controls the media outputdevice to play the requested media content.

In some implementations, the user voice designation of the media outputdevice includes a description of the destination media output device.The voice assistance server 112 or the cloud cast service serveridentifies in the device registry 118 the destination media outputdevice among a plurality of media output devices according to thedescription of the destination media output device. In someimplementations, the description of the destination media output deviceincludes at least a brand (“Samsung TV”) or a location of the mediaoutput device (“my Living Room TV”).

Voice Activated Media Transfer Among Media Output Devices

In some implementations, the operating environment 100 includes a firstcast device (not shown) and a first output device (not shown) coupled tothe first cast device. The operating environment 100 also includes asecond cast device (not shown) and a second output device (not shown)coupled to the second cast device. The cast devices are optionallylocated in the same location (e.g., the living room) or two distinctlocations (e.g., two rooms) in the operating environment 100. Each ofthe cast devices is configured to obtain media content or Internetcontent from content hosts 114 for display on the output device coupledto the respective cast device. Both the first and second cast devicesare communicatively coupled to the voice assistance server 112,optionally a cloud cast service server, and the content hosts 114.

The operating environment 100 further includes one or morevoice-activated electronic devices 190 that are communicatively coupledto the voice assistance server 112 and optionally the cloud cast serviceserver. In some implementations, the voice-activated electronic devices190 are disposed independently of the cast devices and the outputdevices. For example, an electronic device 190 is disposed in a roomwhere no cast device or output device is located. In someimplementations, a first electronic device 190 is disposed in proximityto the first cast device and the first output device, e.g., the firstelectronic device 190, the first cast device and the first output deviceare located in the same room. Optionally, a second electronic device 190is disposed independently of or in proximity to the second cast deviceand the second output device.

When media content is being played on the first output device, a usermay send a voice command to any of the electronic devices 190 to requestplay of the media content to be transferred to the second output device.The voice command includes a media play transfer request. In onesituation, the user can issue the voice command to the electronic device190 disposed in proximity to the first cast device before the user movesto a destination location. Alternatively, in another situation, the usercan issue the voice command to the electronic device 190 disposed inproximity to the second device after the user reaches the destinationlocation.

The voice command is transmitted to the voice assistance server 112. Thevoice assistance server 112 sends a media display information request tothe first cast device to request instant media play information of themedia content that is currently being played on the first output devicecoupled to the first cast device. The first cast device then returns tothe voice assistance server 112 the requested instant play informationincluding at least information of a first media play application (e.g.,YouTube), the media content that is currently being played (e.g., “LadyGaga—National Anthem—Super Bowl 2016”), and a temporal position relatedto playing of the media content. The second cast device then receives amedia display request including the instant play information from thevoice assistance server 112, and in accordance with the instant playinformation, executes the first media play application that controls thesecond output device to play the media content from the temporallocation.

For example, when a music playlist is played on the first output device,the user says “Play on my living room speakers.” The first output devicestops playing the currently played song, and the stopped song resumes onthe living room speakers. When the song is completed, the living roomspeakers continue to play the next song on the music playlist previouslyplayed on the first output device. As such, when the user is movingaround in the operating environment 100, the play of the media contentwould seamlessly follow the user while only involving limited userintervention (i.e., giving the voice command).

Individual Devices Involved in the Operating Environment

FIGS. 2A-2B are block diagrams illustrating an example electronic device190 that is applied as a voice interface to collect user voice commandsin an operating environment (e.g., operating environment 100) inaccordance with some implementations. The electronic device 190,typically, includes one or more processing units (CPUs) 502, one or morenetwork interfaces 504, memory 506, and one or more communication buses508 for interconnecting these components (sometimes called a chipset).The electronic device 190 includes one or more input devices 510 thatfacilitate user input, such as a button 512, a touch sense array 514,and one or more microphones 516. The electronic device 190 also includesone or more output devices 518, including one or more speakers 520,optionally an array of LEDs 522, and optionally a display 524. In someimplementations, the array of LEDs 522 is an array of full color LEDs.In some implementations, an electronic device 190, depending on the typeof device, has either the array of LEDs 522, or the display 524, orboth. In some implementations, the electronic device 190 also includes alocation detection device 526 (e.g., a GPS module) and one or moresensors 528 (e.g., accelerometer, gyroscope, light sensor, etc.).

Memory 506 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. Memory 506, optionally, includes one or more storagedevices remotely located from one or more processing units 502. Memory506, or alternatively the non-volatile memory within memory 506,includes a non-transitory computer readable storage medium. In someimplementations, memory 506, or the non-transitory computer readablestorage medium of memory 506, stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   Operating system 532 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 534 for connecting the electronic        device 190 to other devices (e.g., the server system 140, one or        more cast devices, one or more client devices, one or more smart        home devices, and other electronic device(s) 190) via one or        more network interfaces 504 (wired or wireless) and one or more        networks 110, such as the Internet, other wide area networks,        local area networks (e.g., local network 104), metropolitan area        networks, and so on;    -   Input/output control module 536 for receiving inputs via one or        more input devices and enabling presentation of information at        the electronic device 190 via one or more output devices 518,        including:        -   Voice processing module 538 for processing audio inputs or            voice messages collected in an environment surrounding the            electronic device 190, or preparing the collected audio            inputs or voice messages for processing at a voice            assistance server 112;        -   LED control module 540 for generating visual patterns on the            LEDs 522 according to device states of the electronic device            190; and        -   Touch sense module 542 for sensing touch events on a top            surface (e.g., on touch sensor array 514) of the electronic            device 190;    -   Voice activated device data 544 for storing at least data        associated with the electronic device 190, including:        -   Voice device settings 546 for storing information associated            with the electronic device 190 itself, including common            device settings (e.g., service tier, device model, storage            capacity, processing capabilities, communication            capabilities, etc.), information of one or more user            accounts in a user domain, settings regarding restrictions            when dealing with a non-registered user, and display            specifications associated with one or more visual patterns            displayed by the LEDs 522;        -   Voice control data 548 for storing audio signals, voice            messages, response messages and other data related to voice            interface functions of the electronic device 190;        -   Voice models data 550 for storing voice models or voice            fingerprints corresponding to users in the user domain that            are associated with the electronic device 190;    -   Device leadership module 552 for determining a leader amongst        multiple electronic devices 190 in response to user voice        inputs;    -   Voice model training module 554 for training and generating        voice models or voice fingerprints 550 that can be used to        identify and disambiguate users in the user domain that are        associated with the electronic device 190;    -   Response module 556 for performing instructions included in        voice request responses generated by the voice assistance server        112, and in some implementations, generating responses to        certain voice inputs; and    -   Noise module 558 for determining noise levels around the        electronic device 190 and provide indications of interfering        noise levels and hints to use alternative ways to wake up the        electronic device.

In some implementations, the voice processing module 538 includes thefollowing modules:

-   -   User identification module 560 for identifying and        disambiguating users who provide voice inputs to the electronic        device 190;    -   Hotword recognition module 562 for determining whether voice        inputs include a hotword for waking up the electronic device 190        and recognizing such in the voice inputs;    -   Request recognition module 564 for determining a user request        included in a voice input; and    -   Recognition quality module 566 for determining a measure of the        quality (e.g., a score) of recognition of hotwords and/or        requests in voice inputs.

In some implementations, the noise module 558 includes the followingmodules:

-   -   Recognition analysis module 568 for analyzing how different        levels of noise affect the operations of the voice processing        module 538 (e.g., in particular the hotword recognition module        562 and/or the request recognition module 564) with respect to        the particular electronic device 190;    -   User volume threshold module 568 for determining, for a given        distance from the electronic device 190, highest comfortable        volume levels at which users will utter voice inputs to the        electronic device, 190;    -   Noise determination module 572 for determining a noise level and        a noise profile around the electronic device 190;    -   Noise profiles module 574 for training and generating noise        profiles of noises that interfere with recognition of hotwords        and/or requests in voice inputs; and    -   Noise profiles store 576 for storing noise profiles that have        been determined to be interfering.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 506, optionally, stores a subset of the modulesand data structures identified above. Furthermore, memory 506,optionally, stores additional modules and data structures not describedabove. In some implementations, a subset of the programs, modules,and/or data stored in the memory 506 can be stored on and/or executed bythe server system 140/voice assistance server 112.

In some implementations, one or more of the modules in memory 506described above are part of a voice processing library of modules. Thevoice processing library may be implemented and embedded on a widevariety of devices. An example of a voice processing library isdescribed in U.S. Provisional Patent Application No. 62/334,434,entitled “Implementations for Voice Assistant on Devices,” filed May 10,2016, which is incorporated by reference herein in its entirety.

FIG. 3 is a block diagram illustrating an example server in the serversystem 140 of an operating environment (e.g., operating environment 100)in accordance with some implementations. An example server is one of avoice assistance server 112. The server 140, typically, includes one ormore processing units (CPUs) 702, one or more network interfaces 704,memory 706, and one or more communication buses 708 for interconnectingthese components (sometimes called a chipset). The server 140 couldinclude one or more input devices 710 that facilitate user input, suchas a keyboard, a mouse, a voice-command input unit or microphone, atouch screen display, a touch-sensitive input pad, a gesture capturingcamera, or other input buttons or controls. Furthermore, the server 140could use a microphone and voice recognition or a camera and gesturerecognition to supplement or replace the keyboard. In someimplementations, the server 140 includes one or more cameras, scanners,or photo sensor units for capturing images, for example, of graphicseries codes printed on the electronic devices. The server 140 couldalso include one or more output devices 712 that enable presentation ofuser interfaces and display content, including one or more speakersand/or one or more visual displays.

Memory 706 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. Memory 706, optionally, includes one or more storagedevices remotely located from one or more processing units 702. Memory706, or alternatively the non-volatile memory within memory 706,includes a non-transitory computer readable storage medium. In someimplementations, memory 706, or the non-transitory computer readablestorage medium of memory 706, stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   Operating system 716 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   Network communication module 718 for connecting the server        system 140 to other devices (e.g., various servers in the server        system 140, client devices, cast devices, electronic devices        190, and smart home devices) via one or more network interfaces        704 (wired or wireless) and one or more networks 110, such as        the Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   User interface module 720 for enabling presentation of        information (e.g., a graphical user interface for presenting        application(s) 826-830, widgets, websites and web pages thereof,        and/or games, audio and/or video content, text, etc.) at a        client device;    -   Command execution module 721 for execution on the server side        (e.g., games, social network applications, smart home        applications, and/or other web or non-web based applications for        controlling a client device, a cast devices, an electronic        device 190 and a smart home devices and reviewing data captured        by such devices), including one or more of:        -   a cast device application 722 that is executed to provide            server-side functionalities for device provisioning, device            control, and user account management associated with cast            device(s);        -   one or more media player applications 724 that is executed            to provide server-side functionalities for media display and            user account management associated with corresponding media            sources;        -   one or more smart home device applications 726 that is            executed to provide server-side functionalities for device            provisioning, device control, data processing and data            review of corresponding smart home devices; and        -   a voice assistance application 728 that is executed to            arrange voice processing of a voice message received from            the electronic device 190 or directly process the voice            message to extract a user voice command and one or more            parameters for the user voice command (e.g., a designation            of a cast device or another electronic device 190); and    -   Server system data 730 storing at least data associated with        automatic control of media display (e.g., in an automatic media        output mode and a follow-up mode), including one or more of:        -   Client device settings 732 for storing information            associated with one or more client device, including common            device settings (e.g., service tier, device model, storage            capacity, processing capabilities, communication            capabilities, etc.), and information for automatic media            display control;        -   Cast device settings 734 for storing information associated            with user accounts of the cast device application 722,            including one or more of account access information,            information for device settings (e.g., service tier, device            model, storage capacity, processing capabilities,            communication capabilities, etc.), and information for            automatic media display control;        -   Media player application settings 736 for storing            information associated with user accounts of one or more            media player applications 724, including one or more of            account access information, user preferences of media            content types, review history data, and information for            automatic media display control;        -   Smart home device settings 738 for storing information            associated with user accounts of the smart home applications            726, including one or more of account access information,            information for one or more smart home devices (e.g.,            service tier, device model, storage capacity, processing            capabilities, communication capabilities, etc.);        -   Voice assistance data 740 for storing information associated            with user accounts of the voice assistance application 728,            including one or more of account access information,            information for one or more electronic device 190 (e.g.,            service tier, device model, storage capacity, processing            capabilities, communication capabilities, etc.);        -   User data 742 for storing information associated with users            in the use domain, including users' subscriptions (e.g.,            music streaming service subscriptions, video streaming            service subscriptions, newsletter subscriptions), user            devices (e.g., devices registered in the device registry 118            associated with respective users), user accounts (e.g.,            users' email accounts, calendar accounts, financial            accounts), and other user data; and        -   User voice profiles 744 for storing voice profiles of the            users in the user domain, including, for example, voice            models or voice fingerprints of the users and comfortable            volume level thresholds of the users.    -   Device registration module 750 for managing the device registry        118 coupled to the voice assistance server 112;    -   Device leadership moderator module 780 for moderating device        leadership determinations between the electronic devices 190 in        the operating environment;    -   Noise module 790 for determining noise levels around the        electronic device 190; and    -   Voice processing module 7100 for processing audio inputs or        voice messages collected in an environment surrounding the        electronic device 190.

In some implementations, the voice assistance server 112 is primarilyresponsible for processing of voice inputs and for noise mitigation, andthus one or more of the programs, modules, and data structures in memory506 described above with reference to FIGS. 2A-2B are included inrespective modules in memory 706 (e.g., the programs, modules, and datastructures included with voice processing module 538 are included invoice processing module 7100, and the programs, modules, and datastructures included with noise module 558 are included in noise module790). The electronic device 190 either transmits captured voice inputsto the voice assistance server 112 for processing, or firstpre-processes the voice inputs and transmits the pre-processed voiceinputs to the voice assistance server 112 for processing. In someimplementations, the voice assistance server 112 and the electronicdevice 190 has some shared and some divided responsibilities regardingprocessing of voice inputs and noise mitigation, and the programs,modules, and data structures shown in FIGS. 2A-2B may be included inboth or divided amongst the voice assistance server 112 and theelectronic device 190. Other programs, modules, and data structuresshown in FIGS. 2A-2B (e.g., voice models data 550, voice model trainingmodule 554), or analogues thereof, may also be included in the voiceassistance server 112.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, memory 706, optionally, stores a subset of the modulesand data structures identified above. Furthermore, memory 706,optionally, stores additional modules and data structures not describedabove.

Example Device Leadership Negotiation

FIGS. 4A-4B illustrate an example of a device leadership negotiationbetween multiple electronic devices (e.g., electronic devices 190) inaccordance with some implementations. In an operating environment withmultiple electronic devices 190 (e.g., operating environment 100), whena user in the environment makes a voice input, one or more of themultiple electronic devices 190 detect and receive the voice input. Theelectronic devices 190 determine a leader amongst themselves to respondto the user's voice input and to detect further voice inputs from theuser.

FIG. 4A shows three voice-activated electronic devices 1002-1 thru1002-3 (e.g., electronic devices 190) in an operating environment (e.g.,operating environment 100). They may all be in the same space as a user(not shown), e.g., in the same room. The user makes a voice input (e.g.,an utterance) (not shown), and the three electronic devices 1002-1 thru1002-3 detect and capture a sample of the voice input at varying degreesor levels of quality. The quality of a captured sample of the voiceinput at an electronic device may be affected by one or more factors,including but not limited to surrounding noise, capabilities of thedevice, distance and position of the device relative to the user,position of the device within the space, and so on.

In some implementations, each of the electronic devices determines alevel of quality of the respective sample of the voice input captured bythe electronic device. The quality level may be expressed as a score orsome other value. For example, in FIG. 4A, each of the electronicdevices 1002 has captured a respective sample of a user's voice inputand has determined as score for its respective sample; the sample atelectronic device 1002-1 is scored as 0.98 (on a scale of 0 to 1, with 1corresponding to the highest quality), the sample at electronic device1002-2 is scored as 0.37, and the sample at electronic device 1002-3 isscored as 0.61. In some implementations, the recognition quality module566 (FIG. 2B) at the electronic device 1002 determines the quality levelof the respective captured sample. In some implementations, criteria forevaluating the quality of a voice sample include, for example, loudnessor amplitude of the sample, presence or absence of noise (e.g.,cross-talk, hissing) and corresponding level, presence or absence ofdistortion (e.g., echo) and corresponding level, frequency profile ofthe sample, and so forth.

The electronic devices 1002 communicate their quality scores with eachother. Thus, for example, electronic device 1002-1 receives the qualityscores from electronic devices 1002-2 and 1002-3. Electronic device1002-2 receives the quality scores from electronic devices 1002-1 and1002-3. Electronic device 1002-3 receives the quality scores fromelectronic devices 1002-1 and 1002-2. If all of the electronic devices1002 are on the same local network (e.g., local network 104, FIG. 1),the electronic devices 1002 exchange scores through the local network.For example, an electronic device 1002 may send its quality score to theother electronic devices 1002 by sending multicast messages within thelocal network. If at least one of the electronic devices 1002 is not onthe local network, scores may be sent to server system 140, and thedevice leadership moderator module 780 of the voice assistance server112 distributes the scores to the electronic devices 1002.Alternatively, the device leadership moderator module 780 determineswhich of the devices is to be the leader based on the scores received,and sends the result of the determination to the devices. In someimplementations, the electronic devices 1002 discover each other andtheir locations through the voice assistance server 112 and deviceregistry 118 (e.g., a device 1002 periodically checks in with the voiceassistance server 112 and is informed of other devices associated withthe same user account based on information in the device registry 118).

In some implementations, if just one of the multiple electronic devices1002 captured a sample of the voice input (e.g., the other electronicdevices 1002 were too far away from the user to detect the voice input),the device that captured the voice input sample is determined to be theleader. In some implementations, if an electronic device 1002 failed tocapture a sample of the voice input (e.g., the device was too far awayfrom the user to detect the voice input), the quality score for thatdevice for the voice input is 0.00 or a null score; a device with a“0.00” or null score is eliminated from contention to be the leader.

When an electronic device 1002 receives the scores from the otherelectronic devices 1002, the electronic device compares its qualityscore with the scores from the other devices. If its score is not thebest (e.g., highest) amongst the scores, the electronic device forgosoutputting responses (e.g., responses generated by the server system140) to the voice input (e.g., by powering down or “turning off” thespeaker 520). If its score is the best amongst the scores, theelectronic device outputs responses to the voice input and subsequentvoice inputs; this device is determined to be the “leader.” In someimplementations, the leader also proceeds to detect and process orpre-process subsequent voice inputs from the user (whether it be theleader processing the voice inputs and generating the responses, theleader pre-processing the voice inputs for transmission to the voiceassistance server 112 which generates the responses, or the leadersimply transmitting the voice inputs to the voice assistance server 112which generates the response), and the non-leaders forgo receiving,detecting, and processing subsequent voice inputs from the user.

In some implementations, the leader determination is performed inresponse to a user's voice input to awaken the device (e.g., a voiceinput that includes the hotword). In some implementations, the leaderdetermination may be performed in response to each voice input from theuser or at intervals of a predefined number of voice inputs from theuser (e.g., at every 5th voice input from the user). In this way, theleadership determination may be updated to account for changingconditions (e.g., the user moving around, the noise level around thedevices change, etc.).

In some implementations, the leader determination based on qualityscores may be overridden based on additional criteria. That is, a devicemay be determined to be the leader even if it would otherwise not bedetermined to be the leader based merely on the quality scores. Theadditional criteria may include, for example, a type of request orcommand in the voice input and a state of the device. For example, ifthe voice input includes a request to play or stop playing a video, thenthe request is of particular relevance to an electronic device 1002 thatis capable of playing videos (e.g., a smart TV, a cast device) and isnot particularly relevant to an electronic device 1002 that is anaudio-only device (e.g., sound system, portable speaker). When therequest is of particular relevance to an electronic device 1002, thenthat electronic device 1002 may be determined to be the leader even ifthe quality score for that device is not the highest. In someimplementations, if the request is of particular relevance to multipleelectronic devices in the space, then the one with the best qualityscore amongst the devices to which the request has particular relevanceis determined to be the leader. As another example, if the command inthe voice input is a command to change device state (e.g., “screenoff”), then a device in a state to which the command is applicable(e.g., a smart TV whose screen is currently on) may be determined to bethe leader despite a lower quality score, and a device in a state towhich the command is not applicable (e.g., a smart TV whose screen isalready off) is passed over for leadership. Further, if therequest/command is not of particular relevance to any of the devices1002 (e.g., the devices 1002 are audio-only voice interface devices thatcan communicate with a non-voice-interface cast device through the voiceinterface server 112, and the command is a request to play video via thecast device), the leadership determination is determined based onquality scores as described above. In some implementations, multipleleadership determination criteria (e.g., quality scores, relevance ofcommand, and state of the device, as described above) may be implementedas a multi-step leadership determination (e.g., determine relevance ofthe command and the device state to narrow leadership candidates, thendetermine leader based on quality scores; determine device with highestscore, then check if command relevance or device state criteria apply tothe other devices) or as a weighted determination (e.g., each criterionis accounted for and weighted in a weighted score).

FIG. 4B illustrates a result of the device leadership negotiationamongst the electronic devices 1002 in FIG. 4A. Electronic device 1002-1determines that its quality score is the best amongst the threeelectronic devices 1002, and it assumes leadership for outputtingresponses to the user. Electronic devices 1002-2 and 1002-3 determinethat their quality scores are not the best amongst the three electronicdevices 1002, and power down their speakers 520 and forgo outputtingresponses to the user.

In some implementations, the electronic devices 1002 that are not theleader also power down their microphones 516, and thus do not detectfurther voice inputs from the user; the leader device also is the leaderwith respect to detection and processing of further voice inputs fromthe user. In some implementations, the electronic devices 1002 that arenot the leader maintain power to their microphones 516 and provideassistance re detection of further voice inputs from the user. Forexample, the leader device and the non-leader devices receive and recordrespective samples of a voice input and transmit them to the voiceassistance server 112, which may user the multiple samples together tobetter process the voice input (e.g., aggregate the samples, choose thebest sample for processing, compare the samples to get the best parts ofeach).

In some implementations, the electronic devices 1002 repeat theleadership determination process as described above after each voiceinput from the user. In some implementations, the electronic devices1002 repeat the leadership determination process as described aboveafter intervals of a predefined number of voice inputs (e.g., determinea leader again at every 5th voice inputs or at every 10th voice input).

In some implementations, one or more of the electronic devices 1002output a visual response after a leader is determined (e.g., along withthe audio response to the voice input, the determined leader displays anotification or a predefined LED light pattern indicating that it is theleader).

In some implementations, each of the electronic devices 1002 outputs avisual indication (e.g., a predefined LED light pattern) that it islistening whenever it is listening to a user (e.g., receiving anddetecting the user's voice input), and then just the leader deviceoutputs the response to the voice input per the leadershipdetermination. For example, when a user is speaking a voice input, theelectronic devices 1002-1 thru 1002-3 display the visual indication toindicate that they are listening while receiving the voice input, andafterwards just the leader device 1002-1 outputs the response per theleadership determination.

In some implementations, an electronic device 1002 indicates that it islistening by displaying using a group of LEDs a characteristic visualpattern, such as one of the patterns shown in FIG. 4G of the '566application (duplicated as FIG. 8D below). The visual patterns can bestatic (unchanging) or dynamic (giving the impression of movementthrough changes in the color, brightness, duty cycle, etc. of individualLEDs). In some implementations, an electronic device 1002 indicates thatit is the leader among multiple listening devices by displaying anothercharacteristic visual pattern using its LEDs. Similarly, multiplelistening devices engaged in a negotiation to determine which deviceshould be the leader can indicate that the negotiation is ongoing bydisplaying another characteristic visual pattern using their respectiveLEDs.

Example Process for Multi-User Personalization

FIG. 5 is a flow diagram illustrating a method 1100 of responding to avoice input of a user in accordance with some implementations. Themethod 1100 is implemented at an electronic device 190 with one or moremicrophones, a speaker, one or more processors and memory storing one ormore programs for execution by the one or more processors.

The electronic device 190 receives (1102) via the one more microphones516 a first voice input from a user. The electronic device 190 capturesand records a sample of the first voice input via the microphone 516. Insome implementations, the first voice input is a freeform voice input,in the sense that the voice input need not be strictly limited to apredefined set of words and phrases within a predefined syntax; thevoice input may be a natural language voice input, for example.

The electronic device 190 compares (1104) the first voice input to a setof voice models associated with a plurality of users associated with theelectronic device 190, where the voice models are trained to theelectronic device. The user identification module 560 compares therecorded sample of the first voice input to voice models in voice modelsdata 550. The voice models data 550 include voice models or voicefingerprints of one or more users in the user domain that are associatedwith the electronic device 190 (e.g., users registered to the electronicdevice 190). In some implementations, the voice models are trained bythe voice model training module 554 to the electronic device 190.

Based on the comparing, the electronic device 190 determines (1106) thatthe first voice input corresponds to a first user of the plurality ofusers. For example, the user identification module 560 identifies avoice model in voice models data 550 that best matches the first voiceinput, and in accordance with the identification of the match determinesthat the user speaking the first voice input is the user to which thematching voice model corresponds. In some implementations, the useridentification module 560 also determines a confidence level or someother similar measure of the quality or closeness of the match between avoice model and the voice input, and identifies a match only if thematch is the best and the confidence level is above a predefinedthreshold.

The electronic device 190 presents (1108) a response in accordance withthe determination. In some implementations, the electronic device 190generates the response in accordance with the determination and with thefirst voice input. The response includes an indication of the identityof the identified first user, to let the first user know that he hasbeen identified. The response also includes a response to any voicerequest in the first voice input. For example, in response to a hotwordvoice input to awaken the electronic device 190, the electronic device190 generates and presents (e.g., outputs) a voice greeting thatincludes the first user's name (e.g., “Hello David,” “Good morningMary”). The response may include a voice output and/or instructions toanother device to perform an operation. For instructions to perform anoperation, the indication of the identified user's identify is included,so that the operation is performed with respect to the identified user.For example, if the operation is playing music from a musicsubscription, the device plays music from the identified user's musicsubscription.

In some implementations, the response is an audible voice output, andpresenting the response includes outputting the audible voice output.The electronic device 190 outputs the audible voice output through thespeaker 520.

In some implementations, the electronic device determines a user requestbased on the first voice input, and generates the response in accordancewith the determined user request. In some implementations, generatingthe response in accordance with the determination and with the firstvoice input includes these determining and generating operations. Inaddition to identifying the speaker of the voice input, the electronicdevice 190 determines the user request in the voice input (e.g., a mediaplayback request, a request for news headlines, a request for emails tobe read, etc.), and generates a response that is responsive to therequest (e.g., an audible acknowledgement of the request, an audibleinquiry to the user for more information needed to fulfill the request).

In some implementations, the determined user request includes a requestto the electronic device to awaken, and the response includes an audiblegreeting, where the greeting includes an indication of an identity ofthe determined first user. If the voice input includes a hotword (i.e.,the voice input is a request to the electronic device 190 to awaken),the generated response includes an audible greeting that includes anidentity of the first user (e.g., “Good morning Dave”). The greetingindicates that the electronic device 190 has awakened and indicates theidentity of the user as identified by the electronic device 190.

In some implementations, the determined user request includes a requestfor information, and generating the response in accordance with thedetermined request includes obtaining the requested information withrespect to the first user. If the request is a request for information(e.g., a request to read the user's emails, request to say out loud theuser's upcoming calendar events), the electronic device 190 accesses theidentified user's information (e.g., the user's emails, the user'scalendar) to generate the response. In some implementations, obtainingthe requested information with respect to the determined first usercomprises accessing an account associated with the first user; in orderto obtain the identified user's information for the response, theelectronic device 190 accesses the user's accounts (e.g., the user'semail account, the user's calendar account). At the electronic device190, the user account information may be stored in the voice devicesettings 546.

In some implementations, the electronic device 190 receives a secondvoice input, compares the second voice input to the set of voice models,based on the comparing, determines that the second voice inputcorresponds to a second user of the plurality of users, and presents aresponse in accordance with the determination that the second voiceinput corresponds to the second user. In some implementations, theelectronic device generates the response in accordance with thedetermination and with the second voice input. If a different userassociated with the electronic device 190 speaks a voice input to theelectronic device 190, the electronic device 190 identifies thedifferent user in a similar manner as the identification of the firstuser—comparing the different user's voice input to the voice models andfinding a best match. With the identification of the different user, theelectronic device 190 is able to generate a response based on this newidentification. Thus, the electronic device 190 is able to switchbetween associated users by comparing the voice inputs to the voicemodels and identifying the speaking user based on the comparing,negating a need for the users to explicitly speak a command to switchthe focus of the electronic device 190 from one user to another user.

In some implementations, the electronic device 190 receives a thirdvoice input, compares the third voice input to the set of voice models,based on the comparing, determines that the third voice inputcorresponds to none of the plurality of users, and in accordance withthe determination that the third voice input corresponds to none of theplurality of users, restricts functionality of the electronic device. Ifa user who is not associated with the electronic device 190 speaks avoice input to the electronic device 190, the electronic device 190receives the voice input and attempts to identify the user. As this useris not associated with the electronic device 190 and thus does not havean associated voice model in the voice models data 550, the electronicdevice 190 determines that this user is an unidentified user. For thisunidentified user, the electronic device 190 may restrict itsfunctionality in order to prevent personal or private information frombeing accessed. For example, the electronic device 190 may act on onlyrequests from the unidentified user that involve non-private ornon-personal information (e.g., request for sports scores, news, stocks,weather). In some implementations, the electronic device 190 may denythe unidentified user access to any functionality of the electronicdevice 190 and the voice assistant service (e.g., by ignoring theunidentified user's voice inputs or responding with a responseindicating that the user is not authorized). In some implementations,the restriction of functionality to unidentified users is based onsettings for the device (e.g., voice device settings 546). The deviceowner can set, for example, whether to restrict functionality, forunidentified users, entirely or just to functionality that does notinvolve personal or private information.

In some implementations, the electronic device includes a display, anddisplays on the display an indication of an identity of the determinedfirst user. If the electronic device 190 includes a display 524, theelectronic device 190 may display the identified user's identify (e.g.,display the identified user's name, display a greeting with theidentified user's name, displaying the identified user's picture,avatar, or other image associated with the identified user) on thedisplay 524 to indicate that the electronic device 190 has identified aspeaker of the voice input and to give the user an opportunity to verifythe identification.

In some implementations, the electronic device includes a display, anddisplays on the display a visual user interface associated with thedetermined first user. If the electronic device 190 includes a display524, the electronic device 190 may display a visual user interfaceidentified with the identified user (e.g., a user interface with a themeassociated with the identified user applied). In some implementations,the electronic device 190 displays on the LEDs 522 a pattern associatedwith the identified user.

In some implementations, the electronic device 190 receives the voiceinput (e.g., capture a sample of the voice input), as in step 1102, andsends the voice input to the voice assistance server 112. The voiceassistance server 112 compares the voice input to a set of voice models(e.g., voice models in user voice profiles 744) associated with aplurality of users associated with the electronic device 190, and basedon the comparing, determines the identity of the user who spoke thevoice input, similar to steps 1104 and 1106. The voice assistance server112 generates the response to the voice input, and transmits thegenerated response to the electronic device 190, which outputs theresponse. In some implementations, the voice assistance server 112identifies the user, and transmits the user identification result to theelectronic device 190, which generates and outputs a response inaccordance with the identification result received from the voiceassistance server 112.

In some implementations, the electronic device 190 receives the voiceinput (e.g., capture a sample of the voice input), as in step 1102,performs pre-processing on the voice input, and sends the pre-processedvoice input to the voice assistance server 112. The pre-processing mayinclude identification of the speaking user, as in steps 1104 and 1106,and the identification result accompanies the voice input to the voiceassistance server 112. Alternatively, the user identification isperformed by the voice assistance server 112. The voice assistanceserver 112 receives the pre-processed voice input, identifies thespeaking user if not already performed by the electronic device 190(similar to steps 1104 and 1106), generates the response to the voiceinput, and transmits the generated response to the electronic device190, which outputs the response.

In some implementations, the electronic device 190 receives the voiceinput (e.g., capture a sample of the voice input) and either sends thevoice input to the voice assistance server 112 or performspre-processing and sends the pre-processed voice input to the voiceassistance server 112. The voice assistance server 112 compares thevoice input to a set of voice models (e.g., voice models in user voiceprofiles 744) associated with a plurality of users associated with theelectronic device 190, and based on the comparing, determines theidentity of the user who spoke the voice input, as in steps 1104 and1105. The voice assistance server 112 generates the response to thevoice input and transmits the generated response to the electronicdevice 190, which outputs the response. In some implementations, thevoice assistance server 112 identifies the user, and transmits the useridentification result to the electronic device 190, which generates andoutputs a response in accordance with the identification result receivedfrom the voice assistance server 112.

As described above, the speaking user is identified by matching a voicemodel or voice fingerprint to the voice input. In some implementations,the electronic device 190 is configured to perform a process forcapturing and training voice models or voice fingerprints. For example,when a user is to be associated with the electronic device 190 (e.g.,registered to the device), the electronic device 190 (e.g., voice modeltraining module 554) may initiate and guide the user to be associatedwith the electronic device 190 through a step-by-step process to captureand train a voice model of the user. During the process, the electronicdevice 190 may prompt the user to speak one or more words or phrases(e.g., the hotword, a phrase or sentence that includes multiple distinctphonemes) in order to generate and train the voice model. The voicemodel training module 554 may direct the LED control module 540 toilluminate various visual patterns on the LEDs 522 to indicate to theuser a status of the voice model capture and training process. In someimplementations, the electronic device 190 stores the trained voicemodels in voice models data 550, and optionally transmits the trainedvoice models to the voice assistance server 112 for storage (e.g., inuser voice profiles 744).

In some implementations, a given user has voice models for eachelectronic device 190 with which the user is associated. The voicemodels at the respective devices 190 are trained to the particulardevice, thus accounting for the particular devices and the environmentssurrounding the respective devices.

In some implementations, the voice model training process is performedby the voice assistance server 112 in conjunction with the electronicdevice 190. For example, when a user is to be associated with anelectronic device 190, a voice model training module (not shown) at thevoice assistance server 112 transmits instructions and audio outputs tothe electronic device 190. The electronic device 190 executes theinstructions and output the audio outputs to guide the user through thetraining process similar to the training process performed by the voicemodel training module 554 described above. The electronic device 190captures the user's spoken voice inputs to the training process andtransmits the captured voice inputs to the voice assistance server 112,which generates the voice models and stores them in user voice profiles744. In some implementations, the voice assistance server 112 transmitsto the electronic device 190 voice models of users associated with theelectronic device 190 for storage (e.g., in voice models data 550) andlocal use.

In some implementations, when a voice model is generated and trained fora user, an authentication token is also generated for the user. Theauthentication token authenticates the user to the voice assistanceserver 112. The speaker's voice input is matched to voice modelsassociated with different users. When a matching voice model is foundfor the speaker's voice input, the authentication token for the useraccount associated with the matching voice model is used. In someimplementations, authentication tokens are pre-generated but initiallynot associated with any voice model. The training process associates avoice model trained as a result of the training process to apre-generated authentication token. In some implementations, after theelectronic device 190 identifies the speaker of a voice input, the voiceinput (or the data containing the user request and associated parametersdetermined from the voice input) is transmitted to the voice assistanceserver 112 along with the authentication token associated with thespeaker; the authentication token authenticates the speaker to the voiceassistance server 112. In some implementations, a speaker may beassociated with an authentication token of a user who has grantedpermission to any speaker. For example, a user associated with anelectronic device 190 may configure his account to allow anyone to usehis music service login credentials, so that other users can access anyconnected output device (e.g., speakers) near the user's device 190.

In some implementations, an electronic device 190 identifies thespeaking user by detecting a nearby key device associated with thespeaking user (e.g., using Bluetooth low energy or ultrasonic RF).

In some implementations, a content host 114 generates the audibleresponse to a voice input. For example, the voice input may include arequest for the balance in a bank account associated with the user(e.g., “what is the balance in my Chase account?”). For such a voiceinput, the electronic device 190 or the voice assistance server 112identifies the speaking user, as described above. After identifying theuser and obtaining the bank account information (from, e.g., user data742), the electronic device 190 or the voice assistance server 112transmits a query for the account balance (e.g., through an applicationprogramming interface (API)) to a content host 114 associated with thebank where the bank account is located (e.g., the bank's databasesystem). The bank content host obtains the balance from its databasesand generates an audible response responsive to the request. The bankcontent host transmits the audible response to the electronic device 190or to the voice assistance server 112 (which then transmits the audibleresponse to the electronic device 190), and the electronic device 190outputs the audible response. In this way, transmission and storage ofaccount data in numerical form over networks and on the device isreduced, which promotes security and privacy of user data.

Example Process for Device Leadership Negotiation

FIG. 6 is a flow diagram illustrating a method 1200 of negotiatingdevice leadership among multiple voice interface devices, in accordancewith some implementations. The method 1200 is implemented at anelectronic device (e.g., device 1002-1, 1002-2, or 1002-3) of two ormore electronic devices 190 (electronic devices 1002), where each of thetwo or more electronic devices includes one or more microphones, aspeaker, one or more processors and memory storing one or more programsfor execution by the one or more processors.

The electronic device 190 detects (1202) a voice input. The electronicdevice 190 captures and records a sample of the voice input from a uservia the microphone 516.

The electronic device 190 determines (1204) a quality score for thedetected voice input. The electronic device 190 determines a score,rating, or some other measure of the quality of the captured voice inputsample. The quality of the sample may be affected by multiple factors,including the user's distance from the device, the volume at which theuser spoke the voice input, the surrounding environment and noise, thecapabilities of the device, and so on. In some implementations, criteriafor evaluating the quality of a voice sample include, for example,loudness or amplitude of the sample, presence or absence of noise (e.g.,cross-talk, hissing) and corresponding level, presence or absence ofdistortion (e.g., echo) and corresponding level, frequency profile ofthe sample, and so forth.

The electronic device 190 communicates (1206) the quality score to theother devices of the plurality of electronic devices, and receives(1208) quality scores generated by the other devices of the plurality ofelectronic devices for detection of the voice input by the otherdevices. The electronic device 190 transmits the score for its voiceinput sample to the other devices of the plurality of electronicdevices, and receives scores from the other devices for their respectivesamples of the voice input.

In accordance with a determination that the quality score generated bythe first electronic device is the highest amongst the generated qualityscore and received quality scores for the voice input, the electronicdevice outputs (1210) an audible response and/or a visual response(e.g., a notification or a LED light pattern) to the detected voiceinput, and the other devices of the plurality of electronic devicesforgo outputting an audible response to the detected voice input. Insome implementations, the device with the highest score (or moregenerally, the best score) amongst the devices with respect to the voiceinput optionally pre-processes its respective voice input sample, andtransmits the voice input sample or the pre-processed voice input sampleto the voice assistance server 112, which generates a response, whichincludes audible output (e.g., voice confirmation of the user request,reading out of the user-requested information), to the voice input andtransmits the response to the devices. Alternatively, the device withthe highest score generates the response to the voice input. In eithersituation, the device with the highest score (e.g., device 1002-1)outputs the response, and the other devices (e.g., devices 1002-2 and1002-3), which have the lower scores, do not output the response. Insome implementations, the electronic device that is the leader (e.g.,the device with the highest score in the example here) also is theleader with respect to processing or pre-processing subsequent voiceinputs from the user.

In accordance with a determination that the quality score generated bythe first electronic device is not the highest amongst the qualityscores for the voice input generated by the plurality of electronicdevices, the electronic device 190 forgoes outputting a response to thedetected voice input. If a device determines that its score is not thehighest amongst the devices with respect to the voice input (e.g.,devices 1002-2 and 1002-3), the device does not output a response to thevoice input. For example, the devices with the lower scores power downtheir speakers 520. In some implementations, the devices with the lowerscores also power down their microphones 516 so that just the devicewith the highest score is detecting further voice inputs. In someimplementations, in accordance with the determination that the qualityscore generated by the first electronic device is not the highestamongst the quality scores for the voice input generated by theplurality of electronic devices, the electronic device 190 forgoesoutputting an audible response to the detected voice input, and mayoutput a visual response (e.g., a visual indication that the device wasnot determined to be the leader).

In some implementations, the plurality of electronic devices iscommunicatively coupled through a local network, and the communicatingand receiving are performed through the local network. The electronicdevices 190 are communicatively coupled through a local network (e.g.,local network 104), and may transmit their scores to each other throughthe local network. For example, a device may multi-cast or broadcast itsscore through the local network.

If at least one of the devices is not on the local network, then thedevices may transmit their scores to the voice assistance server 112. Insome implementations, the device leadership moderator module 780transmits the received scores to the devices, and the devices determinewhether their respective scores are the highest. In someimplementations, the device leadership moderator module determines whichscore is the highest and transmits a result of the determination to thedevices. In some implementations, a hub device may include a modulesimilar to the device leadership moderator module 780 and serve in asimilar role.

In some implementations, the quality score includes a confidence levelof detection of the voice input; the quality score is a confidence levelvalue for the voice input sample. In some implementations, the qualityscore includes a signal-to-noise rating of detection of the voice input;the quality score is a signal-to-noise rating or value (e.g.,signal-to-noise ratio) for the voice input sample.

In some implementations, the electronic device 190 recognizes a commandin the voice input, and in accordance with a determination that a typeof the command is related to the first electronic device, outputs anaudible and/or a visual response to the detected voice input. If thecommand or request in the voice input is recognized as one withparticular relevance to a device of the multiple devices that detectedthe voice input, then that device outputs a response to the voice inputand the other devices forgo outputting the response, even if the deviceto which the command has particular relevance does not have the highestscore among the devices with respect to the voice input. Examples ofcommands or requests that have particular relevance include videoplayback requests (e.g., for cast or TV devices) and music playbackrequests (e.g., for audio systems or speaker systems). Additionally, acommand or request may have particular relevance to a device if thecommand or request identifies the particular device that the user wishesto fulfill the request (e.g., the command “Play video on my bedroom TV”has particular relevance to the TV device in the bedroom because thecommand explicitly identifies that device), or the command/requestincludes a device state parameter that is applicable to the currentstate of the device (e.g., a “screen off” command is relevant to adevice whose screen is on and not relevant to a device without a screenor whose screen is off).

In this way, the multiple devices capture respective samples of thevoice input, and a determination as to which sample is best is made.Based on this determination, one of the devices respond to the voiceinput and further voice inputs from the user. In some implementations,the multiple devices repeat the process described above for each voiceinput or at intervals of a predefined number of voice inputs (e.g.,determine the leader again at every 5th voice input or at every 10thvoice input).

In some implementations, the multiple devices detect the voice input andsend their captured voice input samples to the voice assistance server112. The voice assistance server 112 aggregates the samples or selectsthe best quality sample for processing or compares the samples to getthe best parts of each. In some implementations, when the multipledevices are receiving and detecting the voice input, the devices eachdisplay a visual indication (e.g., a predefined LED pattern) that it islistening to the user.

Example Process for Noise Mitigation

FIG. 7 is a flow diagram illustrating a method 1300 of mitigating noisearound a voice interface device in accordance with some implementations.The method 1300 is implemented at an electronic device 190 with one ormore microphones, a speaker, one or more processors and memory storingone or more programs for execution by the one or more processors. Theelectronic device 190 is configured to be awaked by any of a pluralityof affordances, including a voice-based affordance (e.g., a voice inputthat includes a hotword).

The electronic device determines (1302) a current noise profile of anenvironment around the electronic device. The electronic device 190,while it is idle, may run the noise module 558 (e.g., noisedetermination module 572) to determine a noise profile for itssurroundings. In some implementations, determining the current noiseprofile includes determining a noise level (e.g., volume) of the noise,and capturing and processing the ambient noise. The processing mayinclude, for example, determining frequencies and other parameters ofthe ambient noise.

The electronic device determines (1304) whether the noise profileinterferes with the voice-based affordance. The electronic device 190compares the determined noise profile with noise profiles in the noiseprofile store 576 and evaluates the parameters of the noise profile todetermine if the noise profile (i.e., the current noise surrounding theelectronic device 190) interferes with the voice-based affordance.

In accordance with a determination that the noise profile interfereswith the voice-based affordance, the electronic device 190 presents(1306) a hint to a user to use an affordance of the plurality ofaffordances other than the voice-based affordance to awaken theelectronic device. The hint directs the user to use another affordanceother than the voice-based affordance to awaken the electronic device190. In some implementations, the hint is also or otherwise includesindication that the surrounding noise interferes with hotwordrecognition. For example, the electronic device 190 may display apredefined pattern on the LEDs 522 that indicates surrounding noiseinterference with the voice-based affordance and hints to the user thatthe user should use another affordance to wake up the device (e.g.,pressing a button 512, touching a touch sensor array 514, use anotherdevice communicatively coupled to the electronic device 190, such as asmartphone). The indication/hint may be displayed on the display 524 oroutput as sounds or tones through speaker 520.

In some implementations, wherein the voice-based affordance includes aspoken hotword. The voice-based affordance includes the user speakingthe hotword to awaken the electronic device 190, as described above.

In some implementations, determining whether the noise profileinterferes with the voice-based affordance includes determining whetherthe noise profile negatively affects recognition of the hotword in voiceinputs detected by the electronic device. The electronic device 190determines if the surrounding noise (represented by the noise profile)would negatively affect detection/recognition of the hotword by theelectronic device 190 (e.g., is the noise sufficiently loud to drown outthe user's voice input with the hotword, does the noise have a frequencyprofile that would interfere with the user's voice input with thehotword, would the noise distort the user's voice input with thehotword).

In some implementations, determining whether the noise profilenegatively affects recognition of the hotword in voice inputs includescomparing the noise profile to a noise level threshold of a hotwordrecognition process associated with the electronic device. Therecognition analysis module 568 of the electronic device 190 (or thevoice assistance server 112) analyzes the hotword recognition module 562of the electronic device 190 (e.g., when the electronic device 190 isidle) to determine noise thresholds at which noise would cause problemsfor the algorithm or process used by the module. These thresholds aresaved and may be compared to current noise profiles to determine if thecurrent noise profile would pose a problem for the hotword recognitionmodule 562. The analysis of the hotword recognition module is withrespect to the electronic device 190, and thus the threshold may bedifferent between devices.

In some implementations, determining whether the noise profilenegatively affects recognition of the hotword in voice inputs includescomparing the noise profile to a voice volume level threshold, where thevoice volume level threshold is defined based on a predefined distancefrom the electronic device. The user volume threshold module 570 of theelectronic device 190 (or the voice assistance server 112) performsanalyses of the user's voice inputs to determine a maximum comfortablevolume level at which the user will speak voice inputs, with thedetermination based on a predefined distance (e.g., a predefineddistance that is based on a “typical user distance” beyond which voiceinputs attenuate).

In some implementations, determining whether the noise profilenegatively affects recognition of the hotword in voice inputs includescomparing the noise profile to a predetermined set of noise profilesassociated with the electronic device, wherein the noise profiles of thepredetermined set of noise profiles are known to negatively affectrecognition of the hotword by the electronic device in voice inputs. Theelectronic device 190 compares the noise profiles to previous noiseprofiles (e.g., in noise profiles store 576) that interfered withhotword detection.

In some implementations, the electronic device includes a button, andthe affordance of the plurality of affordances other than thevoice-based affordance includes activating the button. The electronicdevice 190 includes a physical push button 512 and the user can awakenthe device by activating the button 512 in lieu of speaking a voiceinput with the hotword.

In some implementations, presenting a hint to a user to use anaffordance of the plurality of affordances other than the voice-basedaffordance to awaken the electronic device includes illuminating thebutton on the electronic device corresponding to a button-basedaffordance of the plurality of affordances. The button 512 may includelighting (e.g., an embedded LED light), and the electronic device 190may hint the user to use the button 512 instead of speaking the hotwordby lighting up the button 512.

In some implementations, presenting a hint to a user to use anaffordance of the plurality of affordances other than the voice-basedaffordance to awaken the electronic device includes outputting apredefined audible sound. The electronic device 190 may hint to the userto use an alternate affordance by outputting a predefined sound or tone(or set of sounds or tones) through the speaker 520.

In some implementations, the electronic device includes a display, andpresenting a hint to a user to use an affordance of the plurality ofaffordances other than the voice-based affordance to awaken theelectronic device includes displaying the hint on a display. Theelectronic device 190 may hint to the user to use an alternateaffordance by displaying a hint message on the display 524.

In some implementations, the electronic device includes one or more LEDlights, and presenting a hint to a user to use an affordance of theplurality of affordances different from the voice-based affordance toawaken the electronic device includes displaying the hint using the oneor more LED lights. The electronic device 190 may hint to the user touse an alternate affordance by displaying a predefined light pattern onthe LEDs 522.

FIGS. 8A and 8B are a front view 400 and a rear view 420 of avoice-activated electronic device 190 in accordance with someimplementations. The electronic device 190 is designed as warm andinviting, and fits naturally in many areas of a home. The electronicdevice 190 includes one or more microphones 402 and an array of fullcolor LEDs 404. The full color LEDs 404 could be hidden under a topsurface of the electronic device 190 and invisible to the user when theyare not lit. In some implementations, the array of full color LEDs 404is physically arranged in a ring. Further, the rear side of theelectronic device 190 optionally includes a power supply connector 408configured to couple to a power supply.

In some implementations, the electronic device 190 presents a clean lookhaving no visible button, and the interaction with the electronic device190 is based on voice and touch gestures. Alternatively, in someimplementations, the electronic device 190 includes a limited number ofphysical buttons (e.g., a button 406 on its rear side), and theinteraction with the electronic device 190 is further based on press onthe button in addition to the voice and touch gestures.

In some implementations, given simplicity and low cost of the electronicdevice 190, the electronic device 190 includes an array of full colorlight emitting diodes (LEDs) rather than a full display screen. A LEDdesign language is adopted to configure illumination of the array offull color LEDs and enable different visual patterns indicatingdifferent voice processing state of the electronic device 190. The LEDDesign Language consists of a grammar of colors, patterns, and specificmotion applied to a fixed set of full color LEDs. The elements in thelanguage are combined to visually indicate specific device states duringthe use of the electronic device 190. In some implementations,illumination of the full color LEDs aims to clearly delineate thepassive listening and active listening states of the electronic device190 among other important states. Placement of the full color LEDscomplies with physical constraints of the electronic device 190, and thearray of full color LEDs could be used in a speaker that is made by athird party original equipment manufacturer (OEM) based on specifictechnology (e.g., Google Assistant).

When the array of full color LEDs is used in a speaker that is made by athird party OEM based on specific technology, the full color LEDs andthe LED design language are configured to fit a corresponding physicaluser interface of the OEM speaker. In this situation, device states ofthe OEM speaker remain the same, while specific visual patternsrepresenting the device states could be varied (for example, the colorsof the full color LEDs could be different but are displayed with similaranimation effects).

In a voice-activated electronic device 190, passive listening occurswhen the electronic device 190 processes audio inputs collected from itssurrounding environment but does not store the audio inputs or transmitthe audio inputs to any remote server. In contrast, active listeningoccurs when the electronic device 190 stores the audio inputs collectedfrom its surrounding environment and/or shares the audio inputs with aremote server. In accordance with some implementations of thisapplication, the electronic device 190 only passively listens to theaudio inputs in its surrounding environment without breaching privacy ofusers of the electronic device 190.

FIG. 8C is a top view of a voice-activated electronic device 190 inaccordance with some implementations, and FIG. 8D shows six visualpatterns displayed by an array of full color LEDs for indicating voiceprocessing states in accordance with some implementations. In someimplementations, the electronic device 190 does not include any displayscreen, and the full color LEDs provide a simple and low cost visualuser interface compared with the a full display screen. The full colorLEDs could be hidden under a top surface of the electronic device andinvisible to the user when they are not lit. Referring to FIGS. 8C and8D, in some implementations, the array of full color LEDs are physicallyarranged in a ring.

Reference has been made in detail to implementations, examples of whichare illustrated in the accompanying drawings. In the detaileddescription above, numerous specific details have been set forth inorder to provide a thorough understanding of the various describedimplementations. However, it will be apparent to one of ordinary skillin the art that the various described implementations may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, circuits, and networks have not been describedin detail so as not to unnecessarily obscure aspects of theimplementations.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first devicecould be termed a second device, and, similarly, a second device couldbe termed a first device, without departing from the scope of thevarious described implementations. The first device and the seconddevice are both types of devices, but they are not the same device.

The terminology used in the description of the various describedimplementations herein is for the purpose of describing particularimplementations only and is not intended to be limiting. As used in thedescription of the various described implementations and the appendedclaims, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

For situations in which the systems discussed above collect informationabout users, the users may be provided with an opportunity to opt in/outof programs or features that may collect personal information (e.g.,information about a user's preferences or usage of a smart device). Inaddition, in some implementations, certain data may be anonymized in oneor more ways before it is stored or used, so that personallyidentifiable information is removed. For example, a user's identity maybe anonymized so that the personally identifiable information cannot bedetermined for or associated with the user, and so that user preferencesor user interactions are generalized (for example, generalized based onuser demographics) rather than associated with a particular user.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages that are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beobvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The implementations were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the implementationswith various modifications as are suited to the particular usescontemplated.

1. (canceled)
 2. A computer-implemented method of determining a leaderdevice of multiple electronic devices, the computer-implemented methodcomprising: detecting, by a microphone of an electronic device of themultiple electronic devices, a voice input; determining, by a processor,a quality score for the voice input that was detected; accessing, by theprocessor, an additional quality score for the voice input that wasdetected by an additional electronic device of the multiple electronicdevices; determining, based on the quality score and the additionalquality score, that the electronic device is the leader device; andbased on the determining that the electronic device is the leaderdevice, outputting, by the electronic device, a response to the voiceinput.
 3. The computer-implemented method of claim 2, whereindetermining that the electronic device is the leader device comprises:determining that the quality score is higher than the additional qualityscore.
 4. The computer-implemented method of claim 2, wherein accessingthe additional quality score comprises: receiving the additional qualityscore from the additional electronic device.
 5. The computer-implementedmethod of claim 2, further comprising: communicating, by the electronicdevice, the quality score to the additional electronic device.
 6. Thecomputer-implemented method of claim 5, wherein the electronic device isconnected to the additional electronic device via a local network, andwherein communicating the quality score comprises: communicating, by theelectronic device, the quality score to the additional electronic devicevia the local network.
 7. The computer-implemented method of claim 2,wherein outputting the response to the voice input comprises:outputting, by the electronic device, an audible response or a visualresponse to the voice input.
 8. The computer-implemented method of claim2, wherein the additional electronic device foregoes outputting anadditional response to the voice input.
 9. The computer-implementedmethod of claim 2, further comprising: transmitting, to a server, thevoice input or a processed version of the voice input, wherein theserver generates the response to the voice input based on the voiceinput or the processed version of the voice input; and receiving, fromthe server, the response to the voice input.
 10. Thecomputer-implemented method of claim 2, wherein the processor isincorporated in the electronic device.
 11. The computer-implementedmethod of claim 2, wherein determining the quality score for the voiceinput comprises: determining, by the processor, the quality score forthe voice input that was detected based on at least one of: a loudnessor amplitude of the voice input, a presence or absence of noise in thevoice input, a presence or absence of distortion in the voice input, ora frequency profile of the voice input.
 12. An electronic devicecomprising: a microphone; an output device; and a processor interfacingwith the microphone and the output device, and configured to: receive,from the microphone, a voice input detected by the microphone, determinea quality score for the voice input that was detected by the microphone,access an additional quality score for the voice input that was detectedby an additional electronic device, determine, based on the qualityscore and the additional quality score, that the electronic device is aleader device, and based on the determining that the electronic deviceis the leader device, cause the output device to output a response tothe voice input.
 13. The electronic device of claim 12, wherein todetermine that the electronic device is the leader device, the processoris configured to: determine that the quality score is higher than theadditional quality score.
 14. The electronic device of claim 12, furthercomprising: a network interface; wherein to access the additionalquality score, the processor is configured to: receive, via the networkinterface, the additional quality score from the additional electronicdevice.
 15. The electronic device of claim 12, wherein the output deviceoutputs the response to the voice input as an audible response or avisual response.
 16. The electronic device of claim 12, furthercomprising: a network interface; wherein the processor is furtherconfigured to: transmit, to a server via the network interface, thevoice input or a processed version of the voice input, wherein theserver generates the response to the voice input based on the voiceinput or the processed version of the voice input, and receive, from theserver via the network interface, the response to the voice input. 17.The electronic device of claim 12, wherein the processor determines thequality score for the voice input based on at least one of: a loudnessor amplitude of the voice input, a presence or absence of noise in thevoice input, a presence or absence of distortion in the voice input, ora frequency profile of the voice input.
 18. A non-transitorycomputer-readable storage medium storing one or more programs, the oneor more programs comprising instructions, which, when executed by anelectronic device, cause the electronic device to perform operationscomprising: detecting, by a microphone, a voice input; determining aquality score for the voice input that was detected; accessing anadditional quality score for the voice input that was detected by anadditional electronic device; determining, based on the quality scoreand the additional quality score, that the electronic device is theleader device; and based on the determining that the electronic deviceis the leader device, outputting, by the electronic device, a responseto the voice input.
 19. The non-transitory computer-readable storagemedium of claim 18, wherein determining that the electronic device isthe leader device comprises: determining that the quality score ishigher than the additional quality score.
 20. The non-transitorycomputer-readable storage medium of claim 18, wherein accessing theadditional quality score comprises: receiving the additional qualityscore from the additional electronic device.
 21. The non-transitorycomputer-readable storage medium of claim 18, wherein the instructionswhich, when executed by the electronic device, cause the electronicdevice to perform operations further comprising: transmitting, to aserver, the voice input or a processed version of the voice input,wherein the server generates the response to the voice input based onthe voice input or the processed version of the voice input; andreceiving, from the server, the response to the voice input.